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We present a theoretical and empirical analysis of a number of bibliometric indicators of journal 
performance. We focus on three indicators in particular, namely the Eigenfactor indicator, the audience 
factor, and the influence weight indicator. Our main finding is that the last two indicators can be 
regarded as a kind of special cases of the first indicator. We also find that the three indicators can be 
nicely characterized in terms of two properties. We refer to these properties as the property of 
insensitivity to field differences and the property of insensitivity to insignificant journals. The empirical 
results that we present illustrate our theoretical findings. We also show empirically that the differences 
between various indicators of journal performance are quite substantial. 

Introduction 

The impact factor (Garfield, 1972, 2006) is without doubt the most commonly 
used bibliometric indicator of the performance of scientific journals. Various 
alternatives to the impact factor have been proposed in the literature. These 
alternatives include indicators based on cited-side normalization (e.g., Van Leeuwen 
& Moed, 2002), indicators based on citing-side normalization (Moed, in press; Zitt & 
Small, 2008), indicators based on the A-index (e.g., Braun, Glanzel, & Schubert, 
2006), and indicators based on recursive citation weighting. Lidicators based on 
recursive citation weighting were first proposed by Pinski and Narin (1976; see also 
Geller, 1978), and they have been popular in the field of economics (Kalaitzidakis, 
Mamuneas, & Stengos, 2003; Kodrzycki & Yu, 2006; Laband & Piette, 1994; 
Liebowitz & Palmer, 1984; Palacios-Huerta & Volij, 2004). The successful PageRank 
algorithm of the Google search engine (Brin & Page, 1998; Page, Brin, Motwani, & 
Winograd, 1998; see also Langville & Meyer, 2006) has caused a renewed interest in 
recursive indicators of journal performance. Three PageRank-inspired indicators that 
have been recently introduced are the weighted PageRank indicator (BoUen, 
Rodriguez, & Van de Sompel, 2006; Dellavalle, Schilling, Rodriguez, Van de 
Sompel, & BoUen, 2007), the Eigenfactor indicator (Bergstrom, 2007; West, 
Bergstrom, & Bergstrom, in press), and the SCImago Journal Rank indicator 
(Gonzalez -Pereira, Guerrero-Bote, & Moya-Anegon, 2009). 

Li this paper, we point out the relation between three indicators of journal 
performance, namely the audience factor (Zitt & Small, 2008), the influence weight 
indicator (Pinski & Narin, 1976), and the Eigenfactor indicator. The audience factor is 
based on citing-side normalization, while the other two indicators are based on 
recursive citation weighting. Unlike the audience factor and the influence weight 
indicator, the Eigenfactor indicator is a parameterized indicator. Hence, the behavior 
of the Eigenfactor indicator depends on the choice of a parameter. Our main flnding is 
that the audience factor and the influence weight indicator can be regarded as a kind 
of special cases of the Eigenfactor indicator. Related to this, we show how the three 
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indicators can be characterized in terms of two properties that we introduce. We refer 
to these properties as the property of insensitivity to field differences and the property 
of insensitivity to insignificant journals. Literestingly, it turns out that the parameter 
of the Eigenfactor indicator can be used to make a trade-off between the two 
properties. In addition to a theoretical analysis of the audience factor, the influence 
weight indicator, and the Eigenfactor indicator, we also report some results of an 
empirical analysis of these indicators. 

This paper is organized as follows. First, we discuss our indicators of interest and 
we point out how these indicators are mathematically related to each other. Next, we 
study the indicators empirically. Finally, we briefly discuss some other related 
indicators and we summarize our conclusions. Some technical details are elaborated 
in an appendix. 

Indicators 

In this section, we discuss the indicators that we study in this paper. We use the 
following mathematical notation. Let there be n journals, denoted by 1, n. Let T\ 
and T2 denote two time periods, where period Ti precedes period T2. (The two periods 
may overlap or coincide.) We are interested in measuring the performance of journals 
1 , ...,n based on citations from articles published in period T2 to articles published in 
period T\. Let aa and a,2 denote the number of articles published in journal i in, 
respectively, periods T\ and T2, and let Cy denote the number of citations from articles 
published in journal i in period T2 to articles published in journal j in period T\. We 
define Si as 



Hence, 5, denotes the total number of citations from articles published in journal i in 
period T2 to articles pubhshed in journals 1, ...,n'm period T\. 

Using the above mathematical notation, we now discuss our indicators of interest. 
We focus on the essential characteristics of the indicators. We ignore practical issues 

such as the document types (e.g., articles, letters, and reviews) that are taken into 
account, the length of the time window within which citations are counted, and the 
way in which self citations are handled. For our present purposes, issues such as these 
are not important. 

Impact factor 

Although the impact factor is not our main interest in this paper, we include it for 
completeness. The impact factor is defined as the average number of citations that a 
journal has received per article (Garfield, 1972, 2006). Hence, the impact factor of 
journal i can be written as 



The impact factor is a very simple indicator. It is well known that in some fields 
articles are on average cited much more frequently than in other fields. The impact 

factor does not correct for such differences among fields. Because of this, impact 
factors of journals in different fields should not be directly compared with each other. 
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Audience factor 

The audience factor is a recent proposal of Zitt and Small (2008). The audience 
factor is similar to the impact factor except that citations are weighted based on the 
journal from which they originate. The larger a journal's average number of 
references per article, the lower the weight of a citation originating from the journal. 
The audience factor of journal i is defined as 



AF,=— X— c.., (3) 

where mj and are given by 

mj=^, (4) 
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m^ = ^' ' , (5) 



that is, nij denotes journal y's average number of references per article and denotes 
the average number of references per article for all journals taken together. Notice that 
in the definitions of mj and only references to articles published in journals 1, ...,n 
in period T\ are taken into account. These references are called active references by 
Zitt and Small. All non-active references are ignored. 

By assigning weights to citations, the audience factor aims to correct for 
differences among fields. Unlike indicators based on cited-side normalization (e.g.. 
Van Leeuwen & Moed, 2002), which also aim to correct for field differences, the 
audience factor has the advantage that it does not rely on an externally imposed field 
classification. In Appendix A, we introduce the property of insensitivity to field 
differences. This property provides a formal definition of the idea of correcting for 
field differences. Informally, the property of insensitivity to field differences has the 
following interpretation. Suppose that we have two equally-sized fields and that each 
journal gives away only a small amount of citations to journals that are not in its own 
field. We then say that an indicator is insensitive to field differences if the average 
value of the indicator for one field deviates from the average value of the indicator for 
the other field only by a small amount. In the case of two fields without any between- 
fields citation traffic, the property of insensitivity to field differences requires that the 
average value of an indicator is the same for both fields. We show in appendix A that 
under a relatively mild assumption the audience factor has the property of 
insensitivity to field differences. 

Influence weight 

The influence weight indicator was proposed by Pinski and Narin (1976). The 
influence weights of journals I, n, denoted by IWi, IW„, are obtained by 
solving the following system of linear equations:^ 



' This system of linear equations has a unique solution if the journal citation matrix C = [c,y] is 
irreducible. In other words, the system of linear equations has a unique solution if in the journal citation 
graph there exists for any two journals i and j a path from i to j and a path from j to i. We note that for 
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iw,. = 



for / = 1, . . . , « 



(6) 



1. 



(V) 



Unlike the impact factor and the audience factor, the influence weight indicator is a 
measure of a journal's average performance per reference rather than of its average 
performance per article. Based on the influence weight of journal i, a measure of 
journal i's average performance per article can be obtained by 



Following Pinski and Narin (1976), we refer to the indicator in Equation 8 as the 
influence per publication indicator. Theoretical studies of the influence weight 
indicator and the influence per publication indicator can be found in papers by Geller 
(1978), Palacios-Huerta and Volij (2004), and Serrano (2004). In the last two papers, 
the influence per publication indicator is referred to as the invariant method. 

In Appendix A, we show that the influence per publication indicator does not have 
the property of insensitivity to field differences. However, the influence per 
publication indicator does have another interesting property, referred to as the 
property of insensitivity to insignificant journals. To see this, consider the following 
example. There are n = 8 journals. Each journal publishes 100 articles in each time 
period. Hence, an = = 100 for / = 1, n. The journal citation matrix C = [c,y] is 
shown in Table 1. Based on this matrix, two fields can be distinguished. One field 
consists of journals 1, 2, 3, and 4. The other field consists of journals 5, 6, 7, and 8. A 
distinction can also be made between fi-equently cited journals and infi-equently cited 
journals. Journals 1, 2, 5, and 6 are frequently cited, while journals 3, 4, 7, and 8 are 
infrequently cited. In practice, it is almost impossible to have publication and citation 
data for all infrequently cited journals in a field. This is because the coverage of 
infi-equently cited journals in bibliographic databases such as Web of Science and 
Scopus is far fi-om complete. Some infi-equently cited journals are covered by these 
databases, but many others are not. To examine the consequences of incomplete 
coverage of infrequently cited journals, we look at two scenarios, scenario 1 and 
scenario 2. In scenario 1, journals 1, 8 are all covered by the bibliographic 
database that we use. In scenario 2, journals 1, 7 are covered while journal 8 is 
not. For both scenarios, influence per publication scores calculated using Equations 6, 
7, and 8 are reported in Table 2. As can be seen in the table, the influence per 
publication scores of journals 1, 7 are very similar in the two scenarios. This 
demonstrates that the influence per publication indicator is rather insensitive to 
incomplete coverage of infrequently cited journals. We therefore say that the 
influence per publication indicator has the property of insensitivity to insignificant 
journals. 



computational reasons it is convenient if the system of linear equations is not only irreducible but also 
aperiodic. 
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TABLE 1 . Journal citation matrix. Rows correspond with citing journals. Columns 
correspond with cited journals. 



Journal 


1 


2 


3 


4 


5 


6 


7 


8 


1 


1000 


1000 


10 


10 


100 


100 


1 


1 


2 


1000 


1000 


10 


10 


100 


100 


1 


1 


3 


1000 


1000 


10 


10 


100 


100 


1 


1 


4 


1000 


1000 


10 


10 


100 


100 


1 


1 


5 


100 


100 


1 


1 


1000 


1000 


10 


10 


6 


100 


100 


1 


1 


1000 


1000 


10 


10 


7 


100 


100 


1 


1 


1000 


1000 


10 


10 


8 


100 


100 


1 


1 


1000 


1000 


10 


10 


TABLE 2. Journals' influence per publicati 


ion scores 


and audience factors. 




Journal 


1 


2 


3 


4 


5 


6 


7 


8 


IPP (scenario 1) 


5.500 


5.500 


0.055 


0.055 


5.500 


5.500 


0.055 


0.055 


IPP (scenario 2) 


5.513 


5.513 


0.055 


0.055 


5.490 


5.490 


0.055 




AF (scenario 1) 


44.000 


44.000 


0.440 


0.440 


44.000 


44.000 


0.440 


0.440 


AF (scenario 2) 


42.938 


42.938 


0.429 


0.429 


34.063 


34.063 


0.341 





What is the relevance of the property of insensitivity to insignificant journals? 
This can be seen as follows. Suppose that instead of the influence per publication 
indicator the audience factor is used in the above example. For both scenario 1 and 
scenario 2, audience factors calculated using Equations 3, 4, and 5 are reported in 
Table 2. Comparing the two scenarios, it is clear that the audience factor does not 
have the property of insensitivity to insignificant journals. Due to the non-coverage of 
journal 8 in scenario 2, journals 5, 6, and 7 have substantially lower audience factors 
in this scenario than in scenario 1. Journals 1, 2, 3, and 4 have only marginally lower 
audience factors. Hence, the non-coverage of journal 8 in scenario 2 causes a 
substantial decrease of the audience factors of journals 5, 6, and 7 relative to the 
audience factors of journals 1, 2, 3, and 4. The results reported in Table 2 demonstrate 
that, when using an indicator that does not have the property of insensitivity to 
insignificant journals, the score of a journal in a certain field may strongly depend on 
the number of infrequently cited journals in the same field that are covered by the 
bibliographic database that one uses. This sensitivity to infrequently cited journals 
may be problematic when comparing scores of journals in different fields. If the 
bibliographic database that one uses covers relatively more infrequently cited journals 
in one field than in another, journals in the former field have an advantage over 
journals in the latter field. 

We have now introduced two properties that bibliometric indicators of journal 
performance may or may not have, namely the property of insensitivity to field 
differences and the property of insensitivity to insignificant journals. It is important to 
note that these two properties rule out each other, that is, an indicator cannot have 
both properties. The following example shows this. Suppose that an infrequently cited 
journal is added to the bibliographic database that one uses. The property of 
insensitivity to insignificant journals then requires that, because the newly added 
journal is infrequently cited, the scores of all other journals remain more or less 
unchanged. The property of insensitivity to field differences, on the other hand, 
requires that the average score of the journals in a field remains unchanged. Hence, in 
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the field to which the newly added journal belongs, the scores of all other journals 
must increase somewhat (otherwise the newly added journal would cause a decrease 
of the average score of the journals in the field). Based on this example, it is clear that 
insensitivity to field differences and insensitivity to insignificant journals are 
confiicting properties that cannot be satisfied both at the same time. 

Eigenfactor 

The Eigenfactor indicator (Bergstrom, 2007; West et al., in press) is a recently 
proposed indicator of journal performance. The indicator belongs to the family of 
PageRank-inspired indicators. Eigenfactor scores of a large number of journals, 
calculated based on Web of Science data, are available at www.eigenfactor.org . 
Eigenfactor scores can also be found in the Journal Citation Reports of Thomson 
Reuters. Various properties of the Eigenfactor indicator are discussed by Franceschet 
(in press-b). Below, we focus on the essential characteristics of the Eigenfactor 
indicator. We ignore the way in which the Eigenfactor indicator handles journal self 
citations and so-caUed dangling nodes. 

The Eigenfactor indicator is a parameterized indicator. Let or e [0, 1] denote the 
parameter of the Eigenfactor indicator. The parameter is similar to what is often 
referred to as the damping factor parameter in the PageRank literature. By default, the 
Eigenfactor indicator uses a equal to 0.85. This is also the default value of the 
damping factor parameter in the PageRank algorithm. Eigenfactor scores are 
calculated as follows (West, Althouse, Rosvall, Bergstrom, & Bergstrom, 2008; West 
& Bergstrom, 2008; for an intuitive description, see West et al., in press). For each 
journal z, a value pi is obtained by solving the following system of linear equations: 

;?,=«X— + (!-«) for/-l,...,« (9) 

Za=i. (10) 

Using the values p\, ...,pn, the Eigenfactor score of journal i is given by 

EF.(a)-100^^^^. (11) 



J 



Unlike for example the impact factor, the Eigenfactor indicator is a measure of a 
journal's total performance rather than of its average performance per article. Hence, 
the Eigenfactor indicator is size dependent. Other things equal, a journal that 
publishes twice as many articles has a twice as high Eigenfactor score. Based on the 
Eigenfactor score of journal i, a measure of journal z's average performance per article 
can be obtained by 

AI,(a) = 5M. ,12) 
lOOa,, 



^ For or < 1, this system of linear equations always has a unique solution. For a~ 1, the system of linear 
equations has a unique solution if the journal citation matrix C = [c,y] is irreducible. 
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The indicator in Equation 12 is referred to as the article influence indicator. 

The properties of the Eigenfactor indicator and the article influence indicator 
depend on the parameter a. We study this dependence in the next section. 

Relation of Eigenfactor with audience factor and influence weight 

In the previous section, formal mathematical definitions of the audience factor, the 
influence weight indicator, and the Eigenfactor indicator were provided. Using these 
definitions, it is relatively easy to see the relation between the three indicators. 
However, we do not compare the indicators directly with each other. This is because 
the indicators are normalized in different ways. That is, the audience factor is a 
measure of a journal's average performance per article, the influence weight indicator 
is a measure of a journal's average performance per reference, and the Eigenfactor 
indicator is a measure of a journal's total performance. In this paper, we focus on 
measuring a journal's average performance per article. Hence, instead of the audience 
factor, the influence weight indicator, and the Eigenfactor indicator, we compare the 
audience factor, the influence per publication indicator, and the article influence 
indicator. As discussed in the previous section, these three indicators all measure a 
journal's average performance per article. 

We first consider the relation between the article influence indicator and the 
audience factor. We assume that the number of articles that journals publish either 
remains stable over time or increases or decreases by the same percentage for all 
journals. Under this assumption, it turns out that, if the Eigenfactor parameter a 
equals 0, the article influence score of a joumal is proportional to the audience factor 
of a journal. Hence, under the above assumption, the audience factor can be regarded 
as a special case of the article influence indicator. This result is stated formally in the 
following theorem. 

Theorem 1, Let the number of articles published in a journal in period Ti be 
proportional to the number of articles published in a joumal in period T\, that is, let aa 
be proportional to aa. Furthermore, let the Eigenfactor parameter a be equal to 0. The 
article influence score of a joumal is then proportional to the audience factor of a 
joumal, that is, AI,(cir) is proportional to AF,. 

Proof. Let a = 0. It then follows from Equations 9, 1 1, and 12 that 

AI,(«)^— X— (13) 
a,, 

It follows from Equations 3 and 4 that 

1 n 

PS,^—Y^-^c.,. (14) 

a,, , s. 

Hence, if a^2 oc , then AI,. (or) oc AF^. . This completes the proof of the theorem. 



^ Two indicators are proportional if they differ by at most a multiplicative constant. For practical 
purposes, indicators that are proportional can be regarded as identical. 
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We now consider the relation between the article influence indicator and the 
influence per publication indicator. It turns out that, if the Eigenfactor parameter a 
equals 1, the article influence score of a journal is proportional to the influence per 
publication score of a journal. Hence, the influence per publication indicator can be 
regarded as a special case of the article influence indicator. This result is stated 
formally in the following theorem."* 

Theorem 2. Let the Eigenfactor parameter a be equal to 1 . The article influence score 
of a journal is then proportional to the influence per publication score of a journal, 
that is, AI,(ci;) is proportional to IPP,. 

Proof. Let a = 1 . It then follows from Equation 9 that 

;^,=Z— • (15) 

Equations 1 1 and 12 then imply that 

AI,(«) = ^. (16) 
Let = IWj.s,. . Equations 6, 7, and 8 can then be rewritten as, respectively, 

5^-1, (18) 

IPP,=^. (19) 

Comparing Equations 10 and 15 to Equations 17 and 18, it is clear that x q.. It 
then follows from Equations 16 and 19 that Al^ (a) oc IPP. . This completes the proof 
of the theorem. 

It follows from Theorems 1 and 2 that the article influence indicator can be 
regarded as a kind of interpolation between the audience factor and the influence per 
publication indicator. The closer the Eigenfactor parameter a is set to 0, the more the 
article influence indicator behaves like the audience factor. The closer the Eigenfactor 



Geller (1978) pointed out that the indicators proposed by Pinski and Narin (1976) can be interpreted 
in terms of Markov chain theory. This is the main insight needed to see the relation between the article 
influence indicator (as well as other PageRank-inspired indicators) and the influence per publication 
indicator. 



8 



parameter a is set to 1, the more the article influence indicator behaves like the 
influence per publication indicator. We know from the previous section that the 
audience factor and the influence per publication indicator have more or less opposite 
properties. Under a relatively mild assumption, the audience factor has the property of 
insensitivity to field differences. The audience factor does not have the property of 
insensitivity to insignificant journals. The influence per publication indicator does 
have the property of insensitivity to insignificant journals but does not have the 
property of insensitivity to field differences. It is now clear that the article influence 
indicator allows one to make a trade-off between the properties of insensitivity to field 
differences and insensitivity to insignificant journals. Setting the Eigenfactor 
parameter a close to gives more weight to the property of insensitivity to field 
differences. Setting the Eigenfactor parameter a close to 1 gives more weight to the 
property of insensitivity to insignificant journals. 

Empirical analysis 

In the previous two sections, indicators of journal performance were studied 
theoretically. We now turn to the empirical analysis of journal performance indicators. 
Like in the previous section, we only consider indicators that measure a journal's 
average performance per article. In addition to the audience factor, the influence per 
publication indicator, and the article influence indicator, we also take into account the 
impact factor. We pay special attention to the effect of the Eigenfactor parameter a on 
the behavior of the article influence indicator.^ For other papers in which PageRank- 
inspired indicators of journal performance are studied empirically, we refer to BoUen 
et al. (2006), BoUen, Van de Sompel, Hagberg, and Chute (2009), Davis (2008), 
Falagas, Kouranos, Arencibia- Jorge, and Karageorgopoulos (2008), Fersht (2009), 
Franceschet (2010, inpress-a), Leydesdorff (2009), Lopez-IUescas, de Moya-Anegon, 
and Moed (2008), and West et al. (2009, in press).^ 

Our empirical analysis is based on the Web of Science database. Only the sciences 
and the social sciences are considered. The arts and humanities are not taken into 
account. We first collected all citations from articles published in 2008 to articles 
published between 2003 and 2007.^ We then selected all journals that have at least 
one incoming citation in each year between 2003 and 2007 and at least one outgoing 
citation in 2008. In this way, we obtained a set of 6708 journals. For each of these 
journals, we calculated six performance measures, namely the impact factor, the 
audience factor, and the article influence score for four different values (0, 0.5, 0.85, 
and 1) of the Eigenfactor parameter a. We did not calculate the influence per 
publication score of a journal. This is because, according to Theorem 2, influence per 
publication scores are perfectly correlated with article influence scores calculated for 
a equal to 1 . In order to keep the analysis as transparent as possible, we calculated all 



^ This is similar to the work of Ding, Yan, Frazho, and Caverlee (2009). Notice, however, that Ding et 
al. study authors rather than journals and that they focus on measuring total performance rather than 

average performance per article. 

^ Davis (2008) and Fersht (2009) compare indicators that measure a journal's total performance. West 
et al. (2009) criticize this approach and explain why it is better to compare indicators that measure a 
joumal's average performance per article. BoUen et al. (2006), Davis (2008), and Franceschet (2010) 
compare indicators of total performance with indicators of average performance per article. It is not 
exactly clear to us how such comparisons should be interpreted. 
^ We only took into account the document types article and review. 

* In addition, we required the journal citation matrix to be irreducible. This also led to the exclusion of 
some journals. 
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performance measures exactly according to the mathematical specification provided 
earlier in this paper. This means that we sometimes deviated slightly from the way in 
which indicators were originally proposed. For example, in the case of the audience 
factor, we did not impose any restrictions on the weight of a citation (unlike Zitt & 
Small, 2008, p. 1859), and in the case of the article influence indicator, we did not 
ignore journal self citations (unlike West et al., 2008, p. 1). 

In Table 3, we report the Pearson and Spearman correlations between the different 
indicators of journal performance. The correlation between the audience factor and 
the article influence indicator for a equal to is of special interest. It follows from 
Theorem 1 that this correlation equals 1 under the assumption that the number of 
articles that journals publish either remains stable over time or increases or decreases 
by the same percentage for all journals. Of course, in practice this assumption does 
not hold exactly. However, as can be seen in Table 3, our empirical results still 
indicate a very high correlation between the audience factor and the article influence 
indicator for a equal to 0. This correlation is also clearly visible in Figure 1, in which 
the empirical relation between the two indicators is shown. Based on Table 3 and 
Figure 1, we conclude that for most practical purposes the two indicators can be 
regarded as identical. 



TABLE 3. Correlations between six indicators of journal performance. Pearson 
correlations are reported in the lower left part of the table. Spearman correlations are 
reported in the upper right part. 





IF 


AF 


AI(O.OO) 


AI(0.50) 


AI(0.85) 


AI(l.OO) 


IF 




0.79 


0.79 


0.82 


0.87 


0.93 


AF 


0.87 




0.98 


0.98 


0.92 


0.77 


AI(O.OO) 


0.88 


0.99 




0.99 


0.93 


0.77 


AI(0.50) 


0.92 


0.98 


0.99 




0.97 


0.84 


AI(0.85) 


0.93 


0.90 


0.91 


0.97 




0.93 


AI(l.OO) 


0.91 


0.76 


0.79 


0.87 


0.96 
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FIG. 1. Relation between the article influence indicator for a equal to and the 
audience factor. 



Another thing to note in Table 3 is the very high correlation between the article 
influence indicator for a equal to and the article influence indicator for a equal to 

0. 5. This correlation is much closer to 1 than the correlation between the article 
influence indicator for a equal to 0.5 and the article influence indicator for a equal to 

1 . Hence, the results presented in Table 3 indicate that for higher values of a the 
article influence indicator is more sensitive to changes in a than for lower values of a. 
For a mathematical explanation for this observation, we refer to Langville and Meyer 
(2006, Section 6.1). 

In Figure 2, we show the empirical relations between four indicators, namely the 
impact factor and the article influence indicator for a equal to 0, 0.85, and 1. 
(Relations for the audience factor and for the article influence indicator for a equal to 
0.5 are not shown. This is because these indicators are both strongly correlated with 
the article influence indicator for a equal to 0.) Although the correlations reported in 
Table 3 are all above 0.75, it is clear from Figure 2 that the relations between most 
indicators are not particularly sfrong. This can be seen even better by excluding the 
journals with the highest scores, as we do in Figure 3. When high scoring journals are 
excluded, relations between indicators can be rather weak. 

For each of the four indicators considered in Figure 2, we list in Table 4 in 
Appendix B the 20 best performing journals. The results shown in Table 4 make clear 
that for high performing journals there can also be substantial differences between 
indicators. Comparing for example the article influence indicator for a equal to and 
the article influence indicator for a equal to 1, there turn out to be only two journals 
that are in the top 10 for both indicators. 



11 




IF Al(0.00) AI(D.S5) AI(I.OO) 




IF AI(D.OO) AI(D.35) AI(I.OO) 




IF AI(D.OO) AI(P.S5) AI(I.OO) 




IF Al(0.00) AI(D.S5) AI(I.OO) 



FIG. 2. Relations between four indicators of journal performance. 
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Article Influence score {ci = 0.00) 

FIG. 3. Relation between the article influence indicator for a equal to and the article 
influence indicator for a equal to 1. Out of the 6708 journals, only the 6252 journals 
with values below 2 for both indicators are shown. 

Our empirical results show that the differences between indicators of journal 
performance are far from negligible. The results also show that the Eigenfactor 
parameter a has a quite large effect on the behavior of the article influence indicator 
(see especially Figure 3). Hence, based on our empirical analysis, it can be concluded 
that the study of different indicators is not merely of theoretical interest but also has a 
substantial practical relevance. We note that some papers (Davis, 2008; Fersht, 2009; 
Leydesdorff, 2009) report strong relations between certain indicators, which may 
seem to contradict our results. However, these papers either focus on indicators of 
total performance, for which it is not surprising to find strong relations (West et al., 
2009), or they rely heavily on Pearson correlation scores. As shown in our analysis, 
high Pearson correlation scores may be somewhat misleading and should be 
interpreted with special care. 

Other indicators related to Eigenfactor 

For completeness, in this section we briefiy consider two other indicators of 
journal performance that are related to the Eigenfactor indicator. These indicators are 
the weighted PageRank indicator proposed by BoUen et al. (2006) and the SCImago 
Journal Rank indicator discussed by Gonzalez-Pereira et al. (2009). SCImago Journal 
Rank scores of a large number of journals, calculated based on Scopus data, can be 
found at www.scimagojr.com . SCImago Journal Rank scores are also reported in the 
Scopus database. Like the Eigenfactor indicator, the weighted PageRank indicator and 
the SCImago Journal Rank indicator belong to the family of PageRank-inspired 
indicators. 



13 



Weighted PageRank scores are obtained by solving the following system of linear 
equations for y9 e [0, 1] and /=0: 



r^=pYp^ + r^^ + (\-P-y)- fori = \,...,n (20) 

(21) 

The weighted PageRank indicator is size dependent and measures a journal's total 
performance. SCImago Journal Rank scores are obtained in two steps. In the first step, 
the above system of linear equations is solved for P, y & [0, 1] and + / < 1.^ By 
default, yff and yoxQ set equal to 0.9 and 0.0999, respectively. '° In the second step, for 
each journal / the SCImago Journal Rank score is calculated by dividing by an. 
Since the SCImago Journal Rank indicator incorporates a normalization for the 
number of articles published in a journal, the indicator measures a journal's average 
performance per article. We note that calculating SCImago Journal Rank scores (or 
weighted PageRank scores) iox fi + y < \ has the effect that smaller journals are 
favored over larger ones. This seems an undesirable effect, and we therefore 
recommend choosing p and / in such a way that P+ y^ 1 . 

Although the weighted PageRank indicator and the SCImago Journal Rank 
indicator seem quite similar to the Eigenfactor indicator, there is a subtle but 
important difference. Choosing p and y in such a way that P + y = \ and solving 
Equations 20 and 21 does not yield Eigenfactor scores. This is due to Equation 11 in 
the calculation of Eigenfactor scores. For this equation, there is no corresponding 
equation in the calculation of weighted PageRank scores or SCImago Journal Rank 
scores. A consequence of this observation is that the behavior of the weighted 
PageRank indicator and the SCImago Journal Rank indicator can be quite different 
from the behavior of the Eigenfactor indicator. For p = I and y=0, a result similar to 
Theorem 2 can be proven, which means that the weighted PageRank indicator and the 
SCImago Journal Rank indicator reduce to the influence per publication indicator of 
Pinski and Narin (1976). However, for P = and y= I, there is no result similar to 
Theorem 1. This means that there is no relation between the SCImago Journal Rank 
indicator and the audience factor of Zitt and Small (2008). In fact, the SCImago 
Journal Rank indicator becomes quite meaningless for p=0 and y= I. The indicator 
simply has the same value for all journals. 

Conclusions 

In a recent report in which research assessment practices based on citation data are 
critically discussed, it is stated that the "assumptions behind (the Eigenfactor 
indicator) are not easy for most people to discern" and that the "complexity (of the 
Eigenfactor indicator) can be dangerous because the final results are harder to 
understand" (Adler, Ewing, & Taylor, 2009, p. 12). These are valid concerns that 
require serious attention. In this paper, we have addressed these concerns by 



^ The SCImago Journal Rank indicator uses different citation windows in the numerator and the 
denominator of the first term in Equation 20. For simplicity, we ignore this issue. 

In an earlier version of the SCImago Journal Rank indicator, the default values of /3 and / were 0.85 
and 0.1, respectively. 
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providing some new insights into the mechanics of the Eigenfactor indicator. Most 
importantly, we have shown the close relation of the Eigenfactor indicator with the 
audience factor (Zitt & Small, 2008) and the influence weight indicator (Pinski & 
Narin, 1976). We have also introduced two properties that bibliometric indicators of 
journal performance may or may not have. These are the properties of insensitivity to 
tield differences and insensitivity to insigniticant journals. Based on the relation 
between the Eigenfactor indicator, the audience factor, and the influence weight 
indicator, we have pointed out that the Eigenfactor indicator (or, more precisely, its 
normalized variant, the article influence indicator) implements a trade-off between 
these two properties. In this way, we have also been able to give a concrete 
interpretation to the parameter of the Eigenfactor indicator. The empirical analysis 
that we have presented has shown that in practice the differences between various 
indicators of journal performance are quite substantial. This further illustrates the 
importance of having a good understanding of the properties of different indicators. 
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Appendix A: The property of insensitivity to field differences 

In this appendix, we introduce the property of insensitivity to field differences. 
We study for different indicators of journal performance whether they have this 
property or not. 

We first introduce the mathematical notation that we use. Suppose two fields can 
be distinguished, field 1 and field 2. There are ni journals in field 1, n2 journals in 
field 2, and n = ni+ ^2 journals in total. Ji = {l, denotes the set of all journals 

in field 1 , J2 = {«i + 1 , • • , «} denotes the set of all journals in field 2, and J=Ji yj J2 
denotes the set of all journals. A denotes a positive matrix of size n x 2. Elements an 
and a,2 of A denote the number of articles published in journal i in, respectively, 
periods Ti and T2. A satisfies 

Hence, the number of articles published in field 1 in period Ti equals the number of 
articles published in field 2 in period T\. C denotes the journal citation matrix. This is 
a non-negative matrix of size n ^ n. Element Cy of C denotes the number of citations 
from articles published in journal i in period T2 to articles published in journal j in 
period T\. 

Using the above mathematical notation, the property of insensitivity to field 
differences can be formally defined as follows. 

Property 1. Let/ denote an indicator of a journal's average performance per article./ 
is said to be insensitive to field differences if and only if 

y. ,a,i/(A,C) y , «ny;(A,C) y aJX^C) 

' <(1 + ^) ^'"^" ' (23) 

for A:= 1,2, for all n\, n2. A, and C, and for all S such that 

Zc,> (1-^)5, (24) 

for A: = 1,2 and for all i g Jk. 

Informally, the property of insensitivity to field differences has the following 
interpretation. Suppose that there are two equally-sized fields and that each journal 
gives away at most a fi-action 5 of its citations to journals that are not in its own field. 
An indicator of journal performance is then said to be insensitive to field differences 
if the average value of the indicator for each field separately deviates no more than a 
fraction d from the average value of the indicator for both fields together. Hence, in 
the case of two fields without any between-fields citation traffic, the property of 
insensitivity to field differences implies that the average value of an indicator is the 
same for both fields. 

It is easy to see that the impact factor is not insensitive to field differences. This is 
not surprising, since it is well known that impact factors of journals in different fields 
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should not be directly compared with each other. The following theorem states that 
under a relatively mild assumption the audience factor of Zitt and Small (2008) is 
insensitive to field differences. 



Theorem 3. Let the number of articles published in a journal in period T2 be 
proportional to the number of articles published in a journal in period T\, that is, let 0,2 
be proportional to aa. The audience factor then is insensitive to field differences. 

Proof. We use the mathematical notation introduced at the beginning of this 
appendix. Let a,2 be proportional to a,i, that is, let there exist a constant 77 > such 
that a,2 - rjua for all i e J. Let ShQ chosen in such a way that Equation 24 is satisfied 
for A: = 1,2 and for all / e Jk. It follows from Equations 3 and 4 that 



"V a,.,AF. V a.^c-Js- 

Equation 24 implies that 



(25) 



i\-5)s.<Y,c.,<s. (26) 



i^Jk 



ifj e Jk and that 



< < (27) 



ifj G J\Jk. Combining Equations 25, 26, and 27 jdelds 

Taking into account that is proportional to an, it follows from Equations 22 and 28 
that 

(1 - S)?jm, < ^ < (1 + S)rpi, . (29) 

It can further be seen that 



/—lie J " 

Equations 29 and 30 imply that 



= Tjms. (30) 
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y a,iAF,. y. , anAF, V a AF,. 

" ' < ^-^^ <(l + S) ^' " ' . (31) 

Hence, Equation 23 is satisfied, which means that the audience factor has Property 1 . 
This completes the proof of the theorem. 

The influence per publication indicator of Pinski and Narin (1976) is not insensitive to 
field differences. This is stated in the following theorem. 

Theorem 4. The influence per publication indicator is not insensitive to field 
differences. 



Proof. We prove the theorem by means of a counterexample. We use the 
mathematical notation introduced at the beginning of this appendix. Let ni = n2 = I, 
let 



A = 



100 100 
100 100 



(32) 



c = 



999 1 
3 997 



(33) 



and let S= 0.003. Equation 24 is then satisfied for A: = 1, 2 and for all i e Jk. It follows 
from Equations 6, 7, and 8 that 



Z. , a. 



(34) 



Hence, Equation 23 is not satisfied, which means that the influence per publication 
indicator does not have Property 1. This completes the proof of the theorem. 

The proof of Theorem 4 illustrates an important problem of the influence per 
publication indicator. When there are two flelds and there is almost no citation traffic 
between the fields, infiuence per publication scores become extremely sensitive to the 
exact number of times one field cites the other (see also West et al., 2008, p. 3). In 
other words, the influence per publication indicator becomes unstable when the 
journal citation matrix is almost reducible. This problem is also discussed by Serrano 
(2004). For a thorough mathematical treatment of this issue for PageRank-inspired 
indicators (of which the influence per publication indicator can be seen as a limit 
case), we refer to Langville and Meyer (2006, Section 6.1). 
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Appendix B: Best performing journals 



TABLE 4. The 20 best performing journals according to four indicators of journal 
performance. 



Journal 


IF 


Joumal 


AI(O.OO) 


Annual Review of Immunology 


37.7 


Reviews of Modem Physics 


16.5 


CA-A Cancer Journal for Clinicians 


37.4 


CA-A Cancer Joumal for Clinicians 


14.7 


Reviews of Modem Physics 


31.8 


New England Joumal of Medicine 


12.0 


New England Journal of Medicine 


29.5 


Annual Review of Immunology 


10.2 


Physiological Reviews 


29.3 


Materials Science & Engineering R- 


9.6 






Reports 




Annual Review of Biochemistry 


27.7 


Physiological Reviews 


9.3 


Nature Reviews Cancer 


27.0 


Chemical Reviews 


9.0 


Nature Reviews Immunology 


26.5 


Annual Review of Biochemistry 


8.4 


Nature Reviews Molecular Cell 


26.5 


Nature Reviews Cancer 


8.0 


Biology 








Annual Review of Neuroscience 


24.9 


Nature Materials 


8.0 


Chemical Reviews 


23.9 


Progress in Materials Science 


7.9 


Cell 


22.3 


Progress in Polymer Science 


7.8 


Annual Review of Cell and 


21.1 


Nature 


7.6 


Developmental Biology 








Nature 


21.1 


J AM A- Joumal of the American 


7.5 






Medical Association 




Nature Reviews Neuroscience 


20.9 


Annual Review of Neuroscience 


7.5 


Nature Immunology 


20.5 


Nature Reviews Molecular Cell 


7.5 






Biology 




Nature Medicine 


20.3 


Science 


7.3 


Science 


20.0 


Nature Reviews Immunology 


7.3 


Nature Genetics 


18.7 


Surface Science Reports 


7.0 


Endocrine Reviews 


18.4 


Annual Review of Fluid Mechanics 


6.5 




Journal 


AI(0.85) 


Joumal 


AI(l.OO) 


Annual Review of Immunology 


22.0 


Annual Review of Immunology 


33.3 


Reviews of Modem Physics 


20.5 


Annual Review of Biochemistry 


27.4 


Annual Review of Biochemistry 


18.2 


Cell 


26.5 


Nature Reviews Molecular Cell 


16.9 


Nature Reviews Molecular Cell 


26.1 


Biology 




Biology 




Cell 


16.8 


Annual Review of Neuroscience 


24.8 


Annual Review of Neuroscience 


16.7 


Annual Review of Cell and 


22.5 






Developmental Biology 




CA-A Cancer Joumal for Clinicians 


16.0 


Nature Reviews Immunology 


21.8 


Nature Reviews Immunology 


14.7 


Nature Immunology 


20.6 


New England Joumal of Medicine 


14.6 


Nature Genetics 


20.0 


Nature 


14.4 


Annual Review of Astronomy and 


19.8 






Astrophysics 




Annual Review of Cell and 


14.3 


Nature 


19.7 


Developmental Biology 








Physiological Reviews 


13.7 


Reviews of Modem Physics 


19.4 


Nature Reviews Cancer 


13.6 


Nature Reviews Cancer 


19.0 


Nature Genetics 


13.4 


Physiological Reviews 


18.8 


Science 


13.3 


CA-A Cancer Joumal for Clinicians 


18.0 


Nature Immunology 


13.1 


Science 


17.3 


Quarterly Joumal of Economics 


12.6 


Nature Reviews Neuroscience 


17.0 


Nature Reviews Neuroscience 


11.8 


New England Joumal of Medicine 


16.9 


Nature Medicine 


10.8 


Nature Cell Biology 


15.3 


Nature Materials 


10.4 


Immunity 


15.2 
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